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AN ELEMENTARY METHOD FOR THE PROBLEM OF COLUMN 
SUBSET SELECTION IN A RECTANGULAR MATRIX 

STEPHANE CHRETIEN AND SEBASTIEN DARSES 


Abstract. The problem of extracting a well conditioned submatrix from any rectangular 
matrix (with e.g. normalized columns) has been a subject of extensive research with appli¬ 
cations to rank revealing factorization, low stretch spanning trees, sparse solutions to least 
squares regression problems, and is also connected with problems in functional and harmonic 
analysis. Here, we provide a deterministic algorithm which extracts a submatrix Xs from 
any matrix X with guaranteed individual lower and upper bounds on each singular value of 
Xs- The proof of our main result is short and elementary. 
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1. Introduction 

Let X G be a matrix such that all columns of X have unit euclidean £ 2 -iiorm. We 

denote by ||x ||2 the t' 2 -norm of a vector x and by ||A|| (resp. HAHjys) the associated operator 
norm (resp. the Hilbert-Schmidt norm). Let Xt denote the submatrix of X obtained by 
extracting the columns of X indexed by T C {1,... ,p}. For any real symmetric matrix A, let 
Afc(A) denote the Uth eigenvalue of A, and we order the eigenvalues as Ai(A) > A 2 (A) > • • •. 
We also write Amin(A) (resp. Amax(^)) for the smallest (resp. largest) eigenvalue of A. We 
finally write IS"! for the size of a set S. 

The problem of well conditioned column selection that we condider here consists in finding 
the largest subset of columns of X such that the corresponding submatrix has all singular 
values in a prescribed interval [1 — e, 1 + e]. The one-sided problem of finding the largest 
possible T such that Aniin(-Al-^A'p) > 1 — e is called the Restricted Invertibility Problem and 
has a long history starting with the seminal work of Bourgain and Tzafriri [1]. Applications of 
such results are well known in the domain of harmonic analysis [1]. The study of the condition 
number is also a subject of extensive study in statistics and signal processing [5]. 

Here, we propose an elementary approach to this problem based on two simple ingredients: 

(1) Choosing recursively y G V, the set of remaining columns of X, verifying 

Q{y) < 

where Q is a relevant quantity depending on the previous chosen vectors; 

(2) a well-known equation (sometimes called secular equation) whose roots are the eigen¬ 
values of a square matrix after appending a row and a line. 

1.1. Historical background. Concerning the Restricted Invertibility problem, Bourgain and 
Tzafriri [1] obtained the following result for square matrices: 
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Theorem 1.1 ([1]). Given a p x p matrix X whose columns have unit i 2 -norm, there exists 

P 

T C {1,... ,p} with \T\ > d -—^ such that C < Aniin(^r^r); where d and C are absolute 
constants. 

See also [4] for a simpler proof. Vershynin [6] generalized Bourgain and Tzafriri’s result to 
the case of rectangular matrices and the estimate of |T| was improved as follows. 

Theorem 1.2 ([6]). Given a nxp matrix X and letting X be the matrix obtained from X by 
i 2 -normalizing its columns. Then, for any e G (0,1), there exists T C {1,. .. ,p} with 

|r| >(i-e)Hk 

such that C'i(e) ^ Amin(^7’^T) ^ Amax(^y^r) ^ C*2(e). 

Recently, Spielman and Srivastava proposed in [3] a deterministic construction of T which 
allows them to obtain the following result. 

Theorem 1.3 ([3]). Let X be a px p matrix and £ G (0,1). Then there exists T C {1, . .. ,p} 
ll-X'lP ll-X'lP 

with \T\ > (1 — eS^—r. —^ such that - < Amin(-^T^T)- 

||A|p p 

The technique of proof relies on new constructions and inequalities which are thoroughly 
explained in the Bourbaki seminar of Naor [2]. Using these techniques, Youssef [7] improved 
Vershynin’s result as: 


Theorem 1.4 ([7]). Given a n x p matrix X and letting X be the matrix obtained from X 
by £ 2 -normalizing its columns. Then, for any £ G (0,1), there exists T C {!,...,p} with 

£2 || J ^||2 

\T\ > — 1 - e < \rain{X\.XT) < Amax(-^T-^r) < 1 + £. 

1.2. Our contribution. We provide a deterministic algorithm that extracts a submatrix T), 
from the matrix X with guaranteed individual lower and upper bounds on each singular value 
of y^. 

Consider the set of vectors Vq = {xi ,..., Xp}, where the Xi are the columns of X. At step 
r = 1, choose yi G Vq. By induction, let us be given yi,...,y^ at step r. Let denote the 
matrix whose columns are yi,... ,yr and let Vk be an unit eigenvector of YfYr associated to 

Xk,r ■= \k{YfYr). 

We say that u{-, ■) satishes the hypothesis (H) if u verifies for r > 1: 

(1.1) 0<u(A:,r) < u{k + l,r + l), A: G {0, • • • , r}; 

(1.2) 0<n(A: + l,r) < u(l,r) < tt(0,r) A G {1, • • • , r — 1}. 

We now introduce the "potential" associated to satisfying (H): 


Qr(x) 


y {vjVxf 
m( 0, r) — u{k, r) ’ 


X G Vo. 
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The following result, for which we propose a short and elementary proof, gives a control on 
all singular values in the column selection problem. 

Theorem 1.5. Let u satisfies Hypothesis (H). Set R < pj^l. Then, we can extract from X 
some submatrices such that for all r and k with 1 < k < r < R, we have 

(1.4) l-^Ru{r-k + l,r)^/\^ < Xk,r < I + Sr u{k,r)^/X^, 


where 



In particular, 


Ai,r < l + 25Ru{l,r). 


2. Proof of Theorem 1.5 

2.1. Suitable choice of the extracted vectors. Consider the set of vectors Vo = {xi,..., Xp}. 
At step 1, choose yi G Vo- By induction, let us be given yi,... ,yr at step r. Let Yr denote 
the matrix whose columns are yi,... ,yr and let be an unit eigenvector of YfYr associated 
to Xk^r ■= S^kiyfYr). Let us choose yr+i G Vr := {a^i,..., Xp} \ {yi,..., yr] so that 


(2-6) E 


{vlYfyr+if 




{viYfxf 


1 ^ 

^E 




u(0, r) — u{k, r) p — r u(0, r) — u{k, r) p — r u(0, r) — u{k, r) 

Lemma 2.1. For all r > 1, yr+i verifies 


u(0, r) — u{k, r) 


Air||Af|p 1 

— —^- sup > —----—- 

p-r -u{k,j) 


Proof. Let Xj. be the matrix whose columns are the x G Vr, i.e. XrX^ = Ylx&Vr Then 
^ = TV {YrVkviY.^XrX^,) < Tv{YrVkviY,^)\\XrX^fi\ < Xk,r\\Xf, 


which yields the conclusion by plugging in into (2.6) since X^^r Y 


2.2. Controlling the individual eigenvalues. It is clear that (1.4) holds for r = 1 since 
then, 1 is the only singular value because the columns are supposed to be normalized. 
Assume the induction hypothesis {Hr)', for all k with 1 < k < r < R, (1.4) holds. 

Let us then show that (Hr+i) holds. By Cauchy interlacing theorem, we have 

Afc+ljr+l ^ S,k,r, 1 Y k Y r 
Afc+l,r+l — ^k+l,r, 0 Y k Y r 1. 

We then deduce, due to the induction hypothesis (Hr) and Assumption (H), 

(2.7) Afc+p,.+i < 1 + SRu{k,r)^/X^ < 1 + SRu{k + l,r + l)^/X^J^, 1 < k < r, 

Afc+i,r+i > 1 - SRu{r - k, r)^/X^ 

(2.8) > 1 - 5Ru{r + 1 - (/c + 1) + 1, r + 1) y^Ai^^+i, 0<k<r-l. 
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It remains to obtain the upper estimate for and the lower one for We write 


(2.9) 


y^+iYr+i 


yi+i 


Vr+l Yj. — 


1 vi+lYr 


Y^^vr+i y/w 


and it is well known that the eigenvalues of are the zeros of the secular equation: 


( 2 . 10 ) 


9(A) := 1-A + ^' 


= 0 . 


We first estimate Ai,r+i which is the greatest zero of q, and assume for contradiction that 
(2.11) Ai,r-+i > 1 + 6ru{ 0, r)^/Xl^. 

From {Hr), we then obtain that for A > 1 + 6Ru{0,r)^Jl^, 

1 


q{X) < 1 - A + 


i'viYryr+i)'^ _ ^ 

/ ^ „./'n „./'c ...A • yy-^)- 


^ «(0, r) - u{k, r) 

Let A'^ be the zero of g. We have 5 ((Ai^,.+i) > g(Ai^r+i) = 0 = ^(A^). But g is decreasing, so 


1 

Al,r+1 < A® = 1 + --_- 


(VkYryr+l)" 


SRy/>^ «(0, r) - u{k, r)' 

Thus, using Lemma 2.1, the equality (1.5) and noting that r < p/2, we can write: 

1 


( 2 . 12 ) + 


< 1 + SRu{0,r)y%^, 


5r p ^^u{0,r) - u{k,r) 

which yields a contradiction with the inequality (2.11). Thus, we have 

(2.13) Ai,r+i < 1 + 6Ru{0,r)y%^ < 1 + 5Ru{l,r + 1)y/Ai,,.+i. 

This shows that the upper bound in (Hr+i) holds. 

Finally, to estimate A,.+i^r+i which is the smallest zero of q, we write 


9(A) > 1 — A — 




^ «(0i r) - u{k, r) 

By means of the same reasonning as above, we show that the lower bound in (iL^+i) holds. 


2.3. Controlling the greatest eigenvalue. Set pi^r = Ai— 1 > 0. 

Since u{l,r) < u{l,R) < u{Q,R), we can write 

Hence, using that x < A^JH\Yx implies x < 2yl, we reach the upper estimate for Ai^^- 
This concludes the proof of Theorem 1.5. 
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3. Two SIMPLE EXAMPLES AND AN OPEN QUESTION 


Let US choose u{k, r) = ^^ 7 ^- Using (r + l)(2r — /c)^ < r(2r + 1 — k)^ and (r + l)(r + /c)^ < 

r(r+ 1 +A:)^, we thus deduce that u verifies Hypothesis (H). Applying Theorem 1.5, we obtain 
that we can extract a submatrix with R columns and < 1 + e, provided that 

-Rlogi? < 


P 


|X| 


which is a slightly weaker bound than the one known from [ 1 ], 

One can also verify that u{k, r) = \/r — k satisfies Hypothesis (H) and yields a similar 
bound. 

An open question is then to know whether there exists a function u satisfying Hypothesis 
(H) and allowing to reach the optimal bound known in the Bourgain Tzafriri theorem [1] via 
our new algorithm. 
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